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SINGULARLY PERTURBED MARKOV CHAINS: LIMIT RESULTS 
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This work focuses on time-inhomogeneous Markov chains with 
two time scales. Our motivations stem from applications in reliability 
and dependability, queueing networks, financial engineering and man- 
ufacturing systems, where two-time-scale scenarios naturally arise. 
One of the important questions is: As the rate of fluctuation of the 
Markov chain goes to infinity, if the limit distributions of suitably 
centered and scaled sequences of occupation measures exist, what 
can be said about the convergence rate? By combining singular per- 
turbation techniques and probabilistic methods, this paper addresses 
the issue by concentrating on sequences of centered and scaled func- 
tional occupation processes. The results obtained are then applied to 
treat a queueing system example. 

1. Introduction. This work focuses on time-inhomogeneous Markov 
chains with two time scales. Our motivations stem from applications in re- 
liability, dependability theory, financial engineering, queueing networks and 
manufacturing systems, where two-time-scale scenarios naturally arise. By 
two time scales, we mean that the systems under consideration involve a 
fast varying time as well as a slowly changing one. A convenient way of 
formulation is to introduce a small parameter e > 0. Then the fast and slow 
times can be represented by t/e and t, respectively. The characteristics of 
the systems associated with the slowly varying time t represent the steady- 
state behavior, whereas those of the fast changing time t/e represent the 
transient behavior. One aims to reduce the complexity of the underlying 
systems by taking advantages of the two-time-scale formulations. 
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Such two-time-scale models have been used in mathematical finance by 
Fouque, Papanicolaou and Sircar [12], in manufacturing systems by Sethi 
and Zhang [28] and in control, optimization and filtering by Kushner [22]. 
The two-time-scale approach may also be adopted to treat reliability and 
dependability; see [10] and references therein for dependability models. In a 
wide variety of situations, one uses time-scale separation to model physical, 
biological and economical phenomena and to highlight the distinct rates of 
variations leading to a rapidly fluctuating Markov chain. To make the com- 
putation manageable and to reduce the complexity, one often uses asymp- 
totic results to devise approximation strategies. 

For example, a challenging and important problem in queueing theory is 
concerned with a time-varying queueing length process. Consider the queue 
length process of an Mt/Mt/l/mo queue, which is a single server queue 
with mo waiting rooms and the corresponding stochastic process is a time- 
inhomogeneous birth-death Markov process. The time- varying nature makes 
the process difficult to handle. Nevertheless, under certain circumstances, 
the time-dependent characteristics at time t may be approximated by their 
quasi-stationary distribution at that time; see [16], [24] and [25]. To achieve 
such an approximation, one of the approaches is to model the underlying 
systems with the help of singular perturbation theory (see, e.g., [2]) resulting 
in a two-time-scale formulation leading to the use of singularly perturbed 
Markov chains. 

Consider a continuous-time Markov chain a e (t) with a countable state 
space N = {1, 2, . . .} or a finite state space M. = {1, . . . , mo}, where e > is 
a small parameter. Suppose that the infinitesimal generator of a e (t) is 



where A(t) and B(t) are themselves generators of certain Markov chains. 
We focus on the convergence rate of a normalized sequence of functional 
occupation measures of a £ (t) for t > 0. When e gets smaller and smaller, 
the Markov chain a s (t) fluctuates more and more rapidly. Within a short 
period of time, the chain will reach its quasi-stationary regime. Thus, we 
can approximate its instantaneous behavior by its quasi-steady-state char- 
acteristics. This brings us to the problem under study with the focus on 
the limit behavior as e — > 0. For real- world applications involving piecewise 
deterministic processes, we refer to [9]; for such processes having two-time- 
scale structures, see [25] for queueing networks, and [28] for manufacturing 
and production planning. 

Owing to its importance in emerging applications, asymptotic properties 
of two-time-scale Markov chains have been studied extensively. When the 
Markov chain a E (t) has a finite state space, under smoothness conditions 
of the generators, Khasminskii, Yin and Zhang [18] constructed asymptotic 
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expansions of probability vectors and transition probability matrices. As- 
suming the fast changing part of the generator consisting of several ergodic 
classes, Yin, Zhang and Badowski [32] showed that a suitably scaled se- 
quence of occupation measures converges weakly to a regime-switching dif- 
fusion that is a system of diffusion processes modulated by a continuous-time 
Markov chain. Yin and Zhang [30] extended the results to the case in which 
the state space is countable. Although convergence of the scaled occupation 
measures has been obtained, the convergence rate has not been determined 
for such singularly perturbed Markov chains in [30]. Nevertheless, in many 
applications, it is crucial to estimate the rate of convergence of functional 
occupation measures to the limit process; see [20] . One of the pertinent ways 
of dealing with this relies on certain strong invariance principles; see [26]. 

There is a vast literature concerning strong invariance principles. Based 
on quantile transforms, Csorgo and Revesz [7] obtained a strong invariance 
principle for partial sums of independent and identically distributed random 
variables. It was subsequently refined by Komlos, Major and Tusnady [19] 
with the best possible error bounds. Csorgo, Horvath and Steinebach [6] 
established strong invariance principles for renewal processes; Csorgo, De- 
heuvels and Horvath [5] treated random stopped sums; Kurtz [23] presented 
a strong approximation theorem for density-dependent Markov chains. Re- 
cently, Csaki, Csorgo, Foldes and Revesz [4] obtained strong approximation 
results for additive functions. 

In this paper, our aim is to establish order of magnitude estimates of the 
rates of convergence of functional occupation measures to the limit processes. 
Our methods rely on the Skorohod representation theorem for martingales 
and asymptotic analysis for moments of singularly perturbed Markov chains. 

The rest of the paper is organized as follows. Section 2 gives the precise 
formulation of the problem together with the assumptions used. Section 3 
is the main part of this paper, which presents the results and the proofs. 
Using the Skorohod representation and asymptotic properties of two-time- 
scale Markov chains, we develop the rates of convergence for sequences of 
scaled and centered functional occupation measures. A queueing application 
example is discussed in Section 4. We conclude the paper in Section 5 with 
additional remarks. 

2. Formulation and preliminaries. Let (fi, T, P) be a probability space. 
By virtue of Lemma 4.4.4 in [8], without loss of generality, we may assume 
that the probability space accommodates all the random variables and pro- 
cesses of our interests. Throughout the paper, we use z' (resp. A') to denote 
the transpose of a vector z (resp. a matrix A), and use 1 to denote an infinite- 
dimensional column vector with all components being 1 [i.e., 1 = (1, 1, . . .)']. 
Henceforth, we write z 1 and a lJ for the ith. component of the vector z and 
the ijth. entry of the matrix A = (a 4 -'), respectively, and use a subscript to 
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index a sequence. In addition, we often use K to denote a generic positive 
constant with the convention KK = K and K + K = K used. 

Let e > and a £ {-) be a time-inhomogeneous Markov chain on (£l,J-, P) 
with countable state space N. Suppose that the infinitesimal generator of 
the chain is given by (1.1). We work with a finite time horizon t E [0, T\. 

Recall that an infinite-dimensional matrix- valued function G(t) = (g i:i (t)) 
defined on [0, T] is a generator of the Markov chain /3(t) if: 

• g lJ (-) is Borel measurable and bounded for each i,j € N; 

• 3 y '(*) > for all i^j; and 

• 2~^5=i 9 lJ (*) = for each i E N, and each bounded and Borel measurable 
function /(•) defined on N, 

/(/?(*)) - / G(s)f(-){f3(s)) ds is a martingale, 
J o 

where 

oo 

G(t)/(-)(0 = E^(*)/(j) for each i G N. 
i=i 

Definition 2.1. A Markov chain j3{t) or its generator G(t) is weakly 
irreducible, if the system of equations 

u(t)l= 1, 

(2.1) 

i/(t)G(t) = 0, 

has a unique solution z/(i) = (v l (t) : i £ N) with z^(t) > for each i G N, where 
= (0,0, ...) is an infinite-dimensional vector. The unique nonnegative 
solution is termed a quasi-stationary distribution. 

Remark 2.2. Note that an equivalent way to write (2.1) is 

u(t)G a (t) = (l!0), 

where G a (t) is an augmented matrix given by G a (t) = (l:G(t)). The defi- 
nition above is an extension of the weak irreducibility given in [18] in that 
it allows the state space of the Markov chains to be countable. Compared 
with the usual notion of irreducibility, it deals with time- varying generators 
and relaxes the usual condition by allowing some components of the quasi- 
stationary distribution to be equal to zero. The motivation stems from a wide 
variety of applications in manufacturing, production planning and queueing 
networks. 
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To proceed, let a probability row vector p e (t) be defined by 

Pe (t) = (pl(t),p 2 £ (t),...,p k ;(t),...) 

(2.2) 

= (P(a £ (t) = l),P(a £ (t) = 2),.. .,P(a £ (t) = k),.. .). 
Then p e (t) is the solution of the forward equation 
p £ (t)=p £ {t)G £ (t), 

OO 

p £ (0)=p(0) satisfying p\0)>0 and 5^p i (0) = l. 

i=i 

For notational simplicity, we have assumed the initial probability vector p(0) 
to be independent of e. 

For countable-state-space Markov chains, working with infinite-dimensional 
vector spaces, it is natural to consider the following linear spaces: 



i\ = < v : v l G R for each iGN and ^ |i>*| < oo 

and 

£oo = : v l G R for each i G N and sup |v l | < oo 
equipped with the norms 

oo 

||f ||i = and ||f||oo = sup 



i=l 



respectively (see [14], page 11). For a linear operator A defined on t\ or 
we use its induced norm = supiun^ ||x^4||, where || • || denotes either the 
t\ norm or the norm. We will use the following conditions. 

(Al) For each t G [0, T], A(t) is weakly irreducible. Moreover, for some n > 0, 
A(.) g C n+2 [0,T] and B(-) G C n+1 [0,T], where C k [0,T] denotes the 
space of functions defined on [0, T] that are /c-times continuously dif- 
ferentiable. 

(A2) There is a k > such that for each real number t > 0, 
(2.4) || eocp(A(0)t) - tv(0)\\oo < Kexp(-Kt), 

where u(t) = (v 1 (t) , v 2 (t) , . . .) is the quasi-stationary distribution cor- 
responding to the generator A(t). 

Remark 2.3. (i) By assuming the generators to be sufficiently smooth, 
we can derive (uniform in the time variable t) asymptotic expansions of the 
probability vectors as well as those of the transition probability matrices. 
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(ii) Assumption (A2) is a Doeblin-type condition; see a similar condition 
in [11], page 217. A condition in a slightly different form is used in [11], 
page 192, in which Doob gave an illuminating discussion. Nevertheless, for 
our purpose of study, the current condition (2.4) appears to be more conve- 
nient. Although time- varying generators and time-inhomogeneous Markov 
chains are considered, (A2) is a condition only concerned with a time- 
invariant generator (or a constant matrix) A(0). Suppose that the Markov 
chain generated by A(Q) is (5{t) and that the corresponding transition proba- 
bility matrix is P(t) = (p ij (i))- Since P(t) = exp(A(0)t), (A2) can be rephrased 
as a condition on the probability distribution of Pit). Relaxing this condi- 
tion using a modified norm for singularly perturbed Markov chains can be 
found in [1]. 

(iii) Note that the discussion up to now is devoted to Markov chains with 
a countable state space. If N is replaced by M, a finite state space, the weak 
irreducibility of Ait) in (Al) implies that (A2) holds. That is, for finite- 
state-space Markov chains, (A2) need not be assumed. Also, in this case, 
the infinite-dimensional v{t) and 1 are replaced by finite-dimensional vectors 
v(t) = (v 1 (t),...,v$ l (t))£R lxmo and 1 = (1, . . . , 1)' G R m ° xl , respectively. 



Let 

(2.5) P e (t ,t) = ^(t ,t)), t>to, 

be the transition matrix with 

p i J(t ,t) = P(ct e (t)=j\a e (t ) = i) for all i.j'eN. 

Define 

=I We{s)=i) ~ V\S). 

For {f(i)}^i G i\, define a sequence of centered functional occupation mea- 
sures by 

z £ (t) = f*(j{ae{s))-f^f{i)A')\ ds 

oo r t 

= £/« / <{*)ds, 
and define a sequence of scaled occupation measures as 

(2-7) Ut) = Z -^- 

Before proceeding further, we present a result on asymptotic expansions of 
the transition probability matrix of the process a £ (-) and a weak invariance 
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principle of £ e (-)> which will be used to prove our main results. General- 
izations of the above results can be found in [30], Theorems 3.7, 3.8 and 
4.4. 

Proposition 2.4. Assume (Al) and (A2). Then the following results 
hold. 

(i) The asymptotic expansions in (2.8) hold uniformly in (to,t) with 
0<t <t<T: 



(2i 



k=0 



k=0 



P 6 (to, t) = £ e k <S> k (t) + £ e k *k ( to, — ) + 0(e 



where $o(t) = l^(t) a^d ^fc(^) ^fcC^o? ' r ) [tfii/i r = (t — io)/ £ ] are ^ e 
solutions of 



$ k (t)A(t) = ^g^-^ 1 (t)B(t), 



k = 1, . . . ,n, 



(2.9) 



rffro(tQ,T) 

*(to,to) = /-*o(<o 



*o(*o,r)-4(t ), 



^fc(tp,r) 



* fc (t ,r)A(*d) + i?fc(to, r), k = l,...,n, 



Vk(to,to) = -$ k (to), 



k-l 



8=0 



r t+l 



i\ dt l 



respectively, where I is the identity matrix. Moreover, for k = 0, 1, . . . , n, 
^fc(-) 6 (jn.+2-k ^ an( ^ ^ ere ex i s t K > and k,q > satisfying ||^ , fc(to) r )||oo < 
if exp(— ko t )- 

(ii) ^4s e — > 0, £ e (-) converges weakly to a diffusion process £(•) suc/i i/wi 

E£(t)=0 and E[£,(t)} 2 = [ a 2 {s)ds with 

Jo 

OO OO I" .QQ 

(2.10) a 2 ( S ) = ££/(z)/(j) ^( S ) / ^>,r)dr 

i=l 7=1 L - 70 



(hi) T/ie following estimate holds: 



(2.11) 



sup 

0<t<T 



^fe(i)] 2 



a 2 (s) ds 



0(e). 
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Remark 2.5. Note that in deriving the weak convergence of £ E (") to 
the diffusion process, a crucial observation is that a mixing condition holds. 
This mixing condition can be stated as follows: For A > 0, denote = 
a{a £ (s) :s>t + A} and use T £ j, = {ct! e (s) : s < t}. For any ./^-measurable 
w and JP"* +A -measurable ? satisfying \w\ < 1 and \q\ < 1, there are K > 
and K\ > such that 

\E{q\F £>t } - Eq\ < Kexp(-KiA/e) a.s., 

(2.12) 

\E[wq] - E[w] ■ E[q]\ < K exp(-KiA/e). 

Condition (2.12) is a consequence of the asymptotic expansions of the 
transition matrices and the exponential decays of the initial layer correc- 
tion terms ^k(') f° r k = 0, . . . ,n, owing to the Doeblin-type condition. This 
mixing property will be used subsequently without specific mentioning. 

When the Markov chain has a finite state space M, define the scaled 
occupation measures £ e (i) similar to the countable state-space counterpart 
but with finite-dimensional vectors used: 

(2.13) l(t) = ^=^w £ (s)ds^jF, 

where w £ (t) = («5*(t)) = (I {as{t)=i} -i/*(t)) E M lxm ° and F = (/(l), . . . , /(mo))' 
is an arbitrary vector in R m ° xl . In this case, statement (i) in Proposition 2.4 
remains the same, and statement (ii) is changed to: £ e (-) converges weakly 
to a diffusion process £(•) such that 

(2.14) E£(t) = and E[^(t)] 2 = /*CT 2 (s)ds, 

Jo 

where 



? 2 w = EE/w/w 

i=ij=i 



/■OO . . POO 

v l (s) / ipQ (s,r)dr + (s) / ^(s,r)dr 
Jo Jo 



see [31], Chapters 4 and 5 for a proof. 

Part (iii) indicates that the second moments of £ e (-) and £(•) differ by 
a small amount. To be more precise, the error bound is of the order 0(e), 
which is also a consequence of the asymptotic expansions. 



Remark 2.6. This paper is devoted to the convergence rate of £ e(i 
to £(■). Why is such a study needed? Although £ e (-) [resp. £ e (-)] has been 
shown to converge to a diffusion process, the rate of convergence is yet to 
be determined. Roughly, the weak convergence of the sequence of functions 
of scaled occupation measures indicates that £ e (-) can be replaced by a 
diffusion process £(■) and the fast variations in £ e (-) can be ignored. However, 
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this result alone does not provide us with further information on to what 
extent we can ignore £ e (-)- To obtain such information and to ascertain 
the convergence rate are the main goals of this paper. As alluded to in the 
Introduction, the result will be obtained by combining singular perturbation 
techniques and strong invariance methods. 

3. A sequence of functional occupation measures. In this section, by 
focusing on £ e (-), the scaled sequence of estimation errors of the centered 
functional occupation measures, we develop asymptotic analysis and ascer- 
tain the rate of convergence of £ e (-) to £(•). 

Theorem 3.1. Assume that conditions (Al) and (A2) hold. Then there 
exist a constant 5 > and a diffusion process £(•) with drift and variance 
satisfying (2.10) such that 

(3.1) sup \Z £ (t)-£(t)\= a . s o(e 5 ). 

0<t<T 

Remark 3.2. The meaning of (3.1) is 

Bm t-")T«') = Q almost surely, 

e^O e 5 

and the limit holds uniformly for t G [0, T] . Henceforth, this notation will be 
used throughout the rest of the paper. 

Theorem 3.1 is stated for a e (t) having a countable state space. If the state 
space is M in lieu of N, the statement of the theorem will be changed as 
follows [see Remarks 2.3(iii) and 2.5]. 

Theorem 3.3. Assume that the Markov chain has a finite state space 
M and (Al) holds. Then there exist a constant 5 > and a diffusion process 
£(•) with drift and variance given by (2.14) such that 

sup \Ut)-m\=a.s.o(e 5 ). 

0<t<T 

Remark 3.4. Theorem 3.1 should be compared with Proposition 2.4. In 
lieu of a weak invariance theorem, a strong invariance principle is obtained 
and the convergence rate is ascertained. It will be seen in the proof that we 
can select any positive number 5 from the interval (0, 1/4). In what follows, 
we will prove Theorem 3.1 only. The proof of Theorem 3.3 can be carried 
out in exactly the same way. 

Proof of Theorem 3.1. To facilitate the presentation, the proof is 
divided into three steps. 
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Step 1. Approximating £ e (t) by a martingale process denoted by M £ {t)/ yfe 
[denned in (3.5)]. The main result in this step is summarized later in Propo- 
sition 3.5. 

To obtain an asymptotic martingale expression for £ e (t), let us define 

Ve{t) = (-f{a £ (i)=l},-f{a £ (t)=2}, • ■ ■ J{a E (t)=k}, ■ ■ ■), 

M £ {t) = rj £ (t) - rj £ (0) - f r) £ ( S )G £ (s) ds. 



Then M £ {t) is an ^^-martingale with T £ ± = a{a £ (s):s < t}. Using the 
result of [15], page 55, we can define a stochastic integral with respect to this 
martingale. Furthermore, the solution of the stochastic differential equation 

dVs(t) = i]e(t)G £ (t) dt + dM £ (t) 

is given by 

%{t) = %{0)Pe(.0,t) + / dM £ (s)P £ {s,t), 

Jo 

where P £ (s,t) is the transition matrix given by (2.5). Noting <J>o(^) = 
from Proposition 2.4(i), we have 

r) E (s) - u(s) = V£ (0)[Pe(0,s) - $ («)] 

(3-2) 

+ f dM £ (r){[P £ (r,s) - $„(*)] 
Jo 

It is easily seen that 
(3.3) 

CxD 

G E (t)f(-)(a E (t)) =E J W(t)=i}G £ (t)/(0W = Ve(t)G e (t)F, 

i=l 

where 

F=(f(l),f(2),...,f(k),...)' satisfying ||F||i<oo. 

Multiplying (3.2) from the right by F together with an integration over [0,t] 
leads to 

(3.4) [\Ti e (8)-u{8)]Fd8- [\ £ (0)[P E (0,s)-$ o (s)]Fds = M £ (t), 
Jo Jo 

where 

(3.5) M E (t) = J* (J* dM E (r){[P E (r, s) - <D (s)] + * (*)}-f) ds. 
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Note that 

(3.6) / / dM £ (r)<f> (s)Fds = / M £ (s)<f> (s)F ds = 0. 

Jo Jo Jo 

Equation (3.6) follows from the observations 

^2 1 {a e (s)=i} = ^ and £-T{a«(0)=i} = l- 

i i 

Thus 

[ V£ {s) - r/ e (0)]$ (s) = Ms) - »»(0)]li/(a) = 0. 
Moreover, for u, s > 0, 

A(it)$ (s) = A(it)li/(a) = 0, 
S(it)*o(«) = B(u)tu(s) = 0. 



As a result, 

M e ( S )$ (s) 



Ve{s) ~ 7? e (0) - / T) e (u)G e (u) du 

Jo 

r] £ (s)<Z> {s) - r ?e (0)$ (s) - / 

JO 



*o(a) 
A(u) 



$o(s) ^ 



= 0. 

Consequently, (3.5) leads to 
(3.7) M e {t) 
Note that 



o Jo 



dM E (r)[Ps(r,s)-$o(s))Fds. 



[r/ £ (s) - is(s)}F ds 



(3.8) 

Next, define 
(3.9) 



t=l 



%(t), ie[0,T]. 



X 6 (t) = f\ £ (0)[P E (0,s) - <S> ( S )]Fds. 
Jo 



By virtue of the asymptotic expansions [see (2.8)], the almost sure bound- 
edness of rj £ (0) and the boundedness of F, we have that for any t, 



\*e(t)\ 1 



(3.10) 



\fe y/e Jo 

K 
< — 



cxp 



(0)[P 6 (0, s) -* (s)]||oo||^||ids 
ds 



V£ Jo 
0(\fe) a.s. 



K0S 

e 
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In view of (2.7), (3.4), (3.8) and (3.10), we have established the following 
result. 



Proposition 3.5. Under the conditions of Theorem 3.1, 



(3.11) 



sup 

0<t<T 



1 



e e (t) - -pM e (t) 



a.s.O(^)- 



In view of the definition of £ e (t) and owing to (3.4) and Proposition 3.5, to 
prove (3.1), it suffices to show that there exists a standard Brownian motion 
W(t) such that 

H 



(3.12) 



sup 

0<i<T 



1 



M e (t)-W[ / a\s)ds 



o{e 6 ) 



a.s. 



for any < 5 < 1/4. Note that 



sup 

0<t<T 



-^M e (t)-W( 



(3.13) 



< max 

l<fc<L^/eJ 



1 



M £ {ek) - W 



+ max max 

l<k<\T/e\ (k-l)e<t<ke 



+ max max 

l<fc<|T/eJ (fe-l)e<t<fe£ 



1 



a 2 (s) ds 
1 



M E (ek) - -==M e {t) 



W 



Ek 



o~ 2 (s) ds 



W 



a (s) ds 



To proceed, the estimate of the terms in the second line of (3.13) is obtained 
in the next step, whereas the last two terms in (3.13) are dealt with in Step 
3. _ 

Step 2. Estimates of \^M e (ek) - W{^ k a 2 (s) ds)\. The result is stated 
in Proposition 3.6. To prove this assertion, we need to establish Proposition 
3.7 first. The proof of Proposition 3.7, in turn, will be done by proving a 
sequence of lemmas. 



Proposition 3.6. For any < 5 < 1/4, 



(3.14) max 

l<fc<|T/ £ J 



-^=M £ (ek) - W 



ek 



cr 2 (s) ds 



ole 



a.s. 



Proof. Define 

M £ , = M £ (0)=0, 

(3.15) M £ , k = M £ (ek) 



for k = 1, 
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Y e , k = M eik -M ejk -. 1 for k = l,... 
From the definition of Y £ k , we have that 
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(3.16) 
and 



Y 



e,l 



JO 



dM £ (r)[P £ (r,s)-<S> (s)]Fds 



rsk p s 

(3.17) Y E:k = / dM £ {r)[P E (r,s)-<f> (s)]Fds for 2<k< 

Je(k-1) JO 



Then {Y £ ^ k , 1 < k < |_irj} is a martingale difference sequence with respect 
to the filtration {J^fc, 1 < k < where T £ ^ k denotes the cr-algebra gen- 

erated by {Y £ j :j < k}. By virtue of the martingale version of the Skorohod 
representation theorem (see [29], Theorem 4.3; also [27]), there exist non- 
negative variables T £jk such that 



(3.18) 



1 



M, k :l<k< 



\w{e{T £ ^ + ---+T £>k )):l<k< 



where D denotes "equal in distribution" and W(-) is a standard Brownian 
motion. Now let Q £ $ be the trivial c-field and let Q ek be the a-field gen- 
erated by !F E>k and : < t < £^i=i T e,i} f° r & — 1- Furthermore, from 
Theorem 4.3 of [29], we have: 

(i) each t £ k is Q £ ^-measurable; 

(ii) eE[r £>1 ] = e- l E[Y £ rf and eE[r £jk ] = e~ l E[Y £>k } 2 ; 

(iii) for each k with 2 < k < [j\ , 

eE{T £tk \g £ik ^} = e~ l E{[Y £!k } 2 \T £tk ^} a.s. 
Using [29], Theorem 4.3 again, there is a positive constant 

L r = 2(8/7r 2 ) r_1 r(r + l), 

where T(-) is the familiar Gamma function such that 

(3.19) e r E[r £A ] r <e- r L r E[Y £A } 2r , 
and for each k with 2 < k < , 

(3.20) e r E{[T £tk } r \G £ . k ^} < e- r L r ^{[y £ifc ] 2 n^ e , fc _i}; 
see [13], page 269. We need to establish the following assertion. 
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Proposition 3.7. For any 9 £ (0, 1/2), 



(3.21) 



max 

l<k<[T/c} 



e^]r £ij - / cr 2 (s)(is 



Suppose momentarily that (3.21) holds. Then 



(3.22) 



sup 

l<k<[T/e\ 



W{e(t £)1 + ■■■ + Te , k )) - W (J* a 2 (s) ds 



< sup sup \W(s + t)-W(s)\. 

0<S< / o T CT 2( u ) du O<t<£« 



It follows from Theorem 1.1.1 of [8] that 



(3.23) 



sup sup \W(s + t)-W(s)\= a . s o{V?). 



0<s< J Q o 2 (u)du 



0<t<e° 



Noting (3.18), (3.14) directly follows from (3.23). Thus it remains to prove 
Proposition 3.7, which is our task in the remainder of this step. □ 

Proof of Proposition 3.7. Note that 

( k r i 

|y^(r £i? - — E{t £ j\Q £ j_i}), I < k < — j is a martingale. 

Using Burkholder's inequality for martingales (see [3], Corollary 1, page 
397), we obtain that for any < 7 < 1, 



P max 

\ l<fc<LT/ej 



3=1 



> 



max 

l<k<[T/e} 



J2( T ^3 ~ E {T B j\Gej-l}) 
3=1 



(3.24) < e ^){i+i) E (^ 

/L'J'/eJ \ 



1+7 



(l+7)/2 



where is a constant depending only on 7. Noting the elementary inequal- 
ity 



(3.25) 



\a + b\ r <C r (\a\ r + \b\ r ), r>0, 
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where C r = 1 or 2 r ~ 1 according to r < 1 or r > 1, (3.24) and (3.25) with 
r < 1 yield that 



P max 

\l<k<[T/e} 



(3.26) 



> e 



< A^ 1 "*)^) ]T E|r eJ - Efcjl&j-i}! 1 ^ for all 7 < 1. 



Note that 7 < 1 is crucial to keep the constant independent of [—J . Using 
Jensen's inequality for conditional expectations, (3.19) and (3.25), 



(3.27) 



< 2V^[£(T ej ) 1+7 + EiE^jg^-t}) 1 ^} 



,1+7 



< 2 1+ V +7 £(r £ii 

< 2 1+ "'L 1+1 £-^E[Y £J } 2+2 \ 



We divide the rest of the proof of Proposition 3.7 into four substeps; each 
of the first three is presented as a lemma. 



Lemma 3.8. 
(3.28) 



i \ 2+27 l T / £ l 
-U £ E\Y e ^ = 0(er). 
v ey k=l 



Proof. Note that rj e (-) is bounded uniformly in t G [0,T], e > 0, and 
the underlying sample point we!] under the norm || • ||oo* That is, 



(3.29) supsup sup 1 1 77 e (t) 1 1 oc < 1. 

uien e>0 0<t<T 

Note also that by Proposition 2.4, in particular by (2.8), 

(3.30) ||P e (0, a )-*oWI|oo = o(e + exp(-^). 
In view of (3.2), for any 1 < k < [T/e\ , 



e,k I 



(3.31) 



e(fe-l) JO 
ek 



dM E (r)[P E (r,s)-® Q (s)]Fds 
r {[(r)e(s) - u(s)]F - Ve(0)[P £ (0, s) - Ms)]F} ds 

Je k-l) 
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< 



ek 

e(fc-l) 
ek 



\rj £ (s) -v(s)\\oo ■ \\F\\ids 



+ 



1^(0)^(0,5) -$ (s)] Hoc -WFWxds 

/e(Jfc-l) ' 

< Ke for some K > 0. 

The constant if above can be chosen to be independent of k and u G by 
virtue of (3.29), (3.30) and the boundedness of This, in turn, implies 

that 

E\Y £tk \ 2+2 ^ <0{e 2+2 ^). 
Substituting the above into the left-hand side of (3.28), we arrive at 

l N 2+2 7 L77fJ 

,7^ 



4= E^) 2+27 



fc=i 



< 0(e-( 1+ T ) )0(e- 1 )0(e 2+27 ) = 0{e>). 
The desired result (3.28) thus follows. □ 

Lemma 3.9. For any 9 G (0, 1/2), 



(3.32) 



max 

l<k<\T/e\ 



a.,.O( £ ). 



PROOF. It follows from (3.24), (3.26)-(3.27) and Lemma 3.8 that 

/ k \ 

eJ^iTsj-EiTejlgsj-!}) >e e ) 

3=1 / 



P max 

\l<k<lT/e] 



(3.33) 

= 0(e- e(1+7) )0(e 7 ) = 0(e e ) with = 7 - 0(1 + 7). 

Note that for any G (0, 1/2), 0/(1 - 0) < 1. Thus, for any G ^0, 1/2), we 
can choose 7 G (0, 1) such that 7 > 0/(1 — 0). This implies that > 0. Next, 
we pick out e n = n~ 2 / 6 . Then from (3.33), 



n=l 



max 

l<k<[T/e n \ 



>e n <oo. 



The Borel-Cantelli lemma implies that 

k 



max 

l<k<[T/e n \ 



3=1 

According to the choice of e n , we have (3.32). □ 



<a,.0(e e n). 
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Lemma 3.10. For any 6 G (0, 1/2), 

(■ fee 



(3.34) max 

X<k<\T/e\ 



eY,E{T e j\g eJ -i}- a 2 {s)ds 



a,0{e 6 



Proof. Using (iii) in Step 2, 



(3.35) 



eY^E{T £j \g E , j - 1 } = -J2E{iYe, j ] 2 \Fe, j -i}. 



i=i 



J'=l 



Note that {V^ &} is an orthogonal sequence (martingale difference). Next we 
claim that 



(3.36) max 

l<k<[T/e] 



e " , 



0(^ 



This can be done by using the same techniques as in the proof of Lemma 3.8. 
We thus omit the details. 

Similarly to M £ ^ in (3.15), define 

£e,0 = £ e (0) = 0, ^ £ k = ^ £ {ek), y s ,k = ie,k -£e,k-l- 

Using the asymptotic equivalence (3.11), we have 

k k 



(3.37) 



max 

i<fc<L T Ad 



y/ e j= x ,=i 



In view of E[J2j =1 Y £d } 2 = £)J =1 E[Y £j ] 2 , (3.37) implies that 



max 

i<fc<L T /<\l 



\j:E[Y e j? - E^yJ 

3=1 \j=l 



O(y^). 



This together with (3.36) and (3.37) implies that 



max 

l<fc<LT/^J 



(3.38) 



£ y2 E {T £ j\g £tj - 1 } - a 2 (s)ds 
i=i ^ 



max 

l<k<[T/e} 



ho 



cj 2 (s) (is 



+ a. S .0(£ e 



where a .s.O(e e ) indicates the order is in the sense of almost sure conver- 
gence. As a result, to derive the desired result, we need only work with 
E(J2j=iEy £ .j) 2 ■ By telescoping and noting that £ £j o = 0> 

2 / k \ 2 



(3.39) 



/ k W / k W 
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Using (2.11), we arrive at 



(3.40) 



max 

l<k<\T/e\ 



E[Uek)Y- a\s)ds 



ks 



0(s) 



Combining the estimates (3.38)-(3.40) obtained thus far, the desired result 
(3.34) then follows. □ 

Completion of the proof of Proposition 3.7. Combining Lemma 3.9 
and Lemma 3.10 yields that for any 9 S (0, 1/2), 



max 

l<fc<L T /eJ 



k pke 
£J2 T e,£- / cr 2 (s)ds 

r~f Jo 



< max 
i<k<[T/s\ 



(3.41) 



+ max 

t<k<\T/e\ 



fc fee: 

e^Eir^g^-x}- [ £ a 2 (s)ds 



= a .s.O(e e ). 
Therefore, we have (3.21). □ 

Step 3. Estimates of the last two terms in (3.13). This is stated in the 
following proposition. 



Proposition 3.11. The following estimates hold: 



(3.42) max max 

l<fc<L T / e J {k-l)e<t<ke 

and 



J-M £ (ek) - 4=M e (t) 
V £ V e 



■■0.8.0W2) 



(3.43) 



max max 

l<k<\T/e\ (k-l)e<t<ke 



W QT* a 2 {s)\ ds-W (j* a 2 (s) ds^j 



PROOF. Similarly to (3.31), 

max max \MJek) - MJt)\ 

l<fc<|T/eJ (k-l)e<t<ke 



max max 

l<fc<LT/eJ (k-l)e<t<ke 



ek 



t JO 



dM £ (r)[P e (r,s)-$ (s)]Fds 
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rek 

= max max / Un F (s)—v(s)]F 
l<k<[T/e](k-l)e<t<ke Jt 

(3.44) - V£ (0)[P £ (0,s)-$ (s)}F}ds 

T] e (s) - v(s)\\oo ■ \\F\\i ds 
+ max I"" 11^(0)^(0,5) -*oW]||do- ll^lli ds 

l<fc<|T/eJ Je(k-l) 

< Ke for some K > 0. 

Thus, (3.42) holds. 

By the boundedness of <r 2 (s), it is easily seen that 
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rek 

< max / 

l<k<[T/e\ Je(k-1) 

rek 



sup 

0<t<T 



E 



a 2 (u)du— / a 2 {u)du 



< Ke. 



k=i J ^ k ~ l ) 

Using [8], Theorem 1.1.1 again yields (3.43). □ 



Completion of the proof of Theorem 3.1. It follows from (3.13), 
Proposition 3.6 and Proposition 3.11 that for any 5 E (0,1/4), 



sup 

0<t<T 



M £ (t)-W[ / a 2 (s)ds 



which, in view of Step 1, implies the theorem. □ 

Remark 3.12. In view of (3.11), (3.13) and (3.42)-(3.43), the rate of 
convergence of the sequence of functional occupation measures to the limit 
process in (3.1) is dominated by the bound of 

rek 



(3.45) 



max 

l<fc<L T /e| 



1 



M £ {ek)-W[ / a\s)ds 



Since we cannot obtain that for any 6 > 1/2, 



(3.46) 



max 

±<k<\T/e\ 



k rke 



a,P{e e 



in view of (3.18) and Theorem 1.1.1 [8], the bound of (3.45) is e s with 
6 G (0,1/4). 

Remark 3.13. Equation (3.21) is the key to obtain our convergence 
rate. This equation is based on inequality (3.20), which is given by (3.18). 
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The relationship (3.18) is an application of Strassen's theorem. Note that 
due to the time-scale separation, the small parameter e > is embedded 
in the processes, which is nonstandard. The new twist is on the use of the 
scaling specific to the Markov chains. 

4. Applications to queueing processes. This section demonstrates how 
the results obtained can be applied to queueing problems, illustrates how 
the asymptotic analysis can help us to gain insight, and provides guidelines 
for treating a class of time-dependent queueing models. One of the main 
ideas is: If the rate of change is slow enough, we can approximate the time- 
inhomogeneous behavior by that of quasi-stationary characteristics leading 
to a substantial reduction of complexity. 

Consider an Mt/Mt/l/mo queue with uiq waiting rooms and the first-in 
first-out service discipline. Let a(t) represent the number of customers in 
the system at time t. The arrival rates are A(i)Aj for < i < mo, and the 
service rates are n(t)fii for 1 < i < mo, where A(t), Aj and fi{ are all 

nonnegative. Then a{t) is a Markov process with the generator G{t) given 
by 

--\(t)\ X(t)X 
M*W -[A(t)Ai+/i(*)Mi] A(t)Ai 

fj,(t)[i mo -i - [A(t)A mo _i + /x(t)/j mo _i] A(t)A mo _i 

For an initial time to £ [0,T], let P(to,t) with t > to be the transition 
matrix (p %:, (to,t)) with 

p ij (t ,t) = P(a(t) =j\a(t )=i) for all < i,j < m . 

Then we have the following system of birth-death equations: 

(4.1) ^-P(t ,t) = P(t ,t)G(t). 

at 

(In the above and henceforth, we add to the state space to include the 
possibility that the queue might be empty, so M. = {0, ...,mo}. All the 
results obtained in the previous sections carry over.) 

Assume that the rate of change of the generator G(t) varies very slowly in 
time so that the process P(to,t) can achieve equilibrium before there is any 
significant change in the rate. Following [24] and [25], we replace the arrival 
rate A(t)Aj and service rate fJ>(t)fii by A(t)Aj/e and fj,(t)ni/e, respectively, 
where e > is a smaller parameter. Also index the queue length process 
by e, and write a(t) = a £ (t). Then the corresponding P e (to,t) satisfies the 
following system of equations: 

(4.2) ^-P £ (to,t) = -P £ (t ,t)G(t). 
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Suppose that A(-) G C 2 [0,T] and //(•) G C 2 [0,T]. Then the generator G(t) 
satisfies (Al). By Proposition 2.4, we can obtain the probability of a e (t) = i 
for any time t, that is, Corollary 5.2 of [25]. Furthermore, we consider the 
occupation time of a £ {-) in a given state during a finite interval time [0,t], 
which is an important performance measure for the system. To this end, let 



( V (ty 



with 

,i m _/-i(<)V'n /v^C'Vtt At 



j = 0, ...,m , 



and 

*o(*o,*) = (V4 J (*o,t)) (mo+1)x(mo+1) = [J - $ (*o)]exp(G(0)* 

Our previous result on weak invariance reveals that £ £ (-) converges weakly 
to a diffusion process ^(-), where ^ e (-) and £(•) are given in Remark 2.5. Thus, 
loosely, ^ e (-) can be replaced by the diffusion process. Nevertheless, the weak 
convergence alone does not provide us with any information on how close 
the approximation is. The needed information is provided in this paper. By 
virtue of Theorem 3.3, for any i with < i < mo, by choosing F = ej the 
standard unit vector in R( ?n o+ 1 ) xl 5 we obtain 

f a I{a e (s)=i} ds = a . s J u\s) ds + sfiW (J a 2 (s) db) + o(e l ' 2+& ), 

Se (0,1/4), 

where W(-) is a standard Brownian motion, and 

roc 

a 2 {s) = 2v i {s) / ^(s,r)dr. 



o 

Thus when we approximate the occupation measure 

t rt 

I{a s (s)=i}ds by / u l {s)ds, 
o Jo 

the approximation error is a y/e perturbation of the standard Brownian 
motion. Hence, the convergence rate result will allow us to further evaluate 
how good the approximation is. 



22 



G. YIN AND H. ZHANG 



5. Concluding remarks. This work has been devoted to limit results of 
scaled sequences of centered functional occupation measures. Under the 
framework of two-time-scale formulation, using the Skorohod representa- 
tion and asymptotic properties of recently developed singularly perturbed 
Markov chains, we have obtained convergence rate theorems, which provide 
almost sure invariance principles leading to the weak convergence rates of 
sequences of the underlying occupation measures. 

The proofs of results use the Skorohod representation in an essential way; 
the results are most suitable for the functional form of occupation mea- 
sures. Several problems are of interest from theoretical as well as practical 
considerations. 

First, in lieu of one-dimensional functional occupation measures, we may 
consider vector-valued occupation processes. Somewhat different techniques 
are required; see, for example, Kiefer [17] and Kuelbs [21] among others. 
Second, the current problem setup is under the stipulation that the fast 
changing part of the generator corresponds to a Markov chain having a single 
ergodic class. A generalization of this requires the consideration that the fast 
varying generator consists of multiple ergodic classes. Many applications lead 
to such Markovian models. The corresponding weak convergence result to 
diffusions with regime switching has been obtained in our recent work [30]; 
the associated convergence rate results will be equally important. However, 
from one ergodic class to multiple ergodic classes is not a straightforward 
extension. One of the difficulties is that the scaled sequence £ e (i) for the 
multiple-ergodic-class case is no longer ^-mixing. The limit process is no 
longer a diffusion process, but rather a switching diffusion. For each fixed t, 
the limit distribution is not purely Gaussian, but rather a Gaussian mixture. 
The technical details need to be thoroughly investigated, and the questions 
require further thought and careful consideration. 
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